Why Maximum Entropy? A Non-axiomatic Approach

Ill-posed inverse problems of the form y = Xp where y is J-dimensional vector of a data, p is m-dimensional probability vector which can not be measured directly and matrix X of observable variables is a known J ×m matrix, J < m, are frequently solved by Shannon’s entropy maximization (MaxEnt, ME). Several axiomatizations were proposed (see for instance [1], [2], [3], [4], [5], [6], [7], [8], as well as [9] for a critique of some of them) to justify the MaxEnt method (also) in this context. The main aim of the presented work is two-fold: 1) to view the concept of complementarity of MaxEnt and Maximum Likelihood (ML) tasks introduced at [10] from a geometric perspective, and consequently 2) to provide an intuitive and non-axiomatic answer to the ’Why MaxEnt?’ question. INTRODUCTION The concept of complementarity of maximum entropy and maximum likelihood tasks, proposed at [10], is in this vignette interpreted from a geometrical point of view of vectors. Two key notions are introduced: collinearity and coherence. In addition to shaping the complementarity into an elegant form the collinearity/coherence concepts offer an elementary answer to the persistent ’Why MaxEnt?’ question. COLLINEARITY AND COHERENCE: ML AND MAXENT Definition 1. System of events {A1,A2, . . . ,Am} is equivalently described by its distribution p = [p1,p2, . . . ,pm], or by its potential u = [u1,u2, . . . ,um]. The relationship of equivalence is